<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
  <title>Boost Test Policies and Protocols</title>
  <meta http-equiv="Content-Type" content="text/html; charset=us-ascii" />
  <link rel="icon" href="/favicon.ico" type="image/ico" />
  <link rel="stylesheet" type="text/css" href=
  "../style-v2/section-development.css" />
  <!--[if IE 7]> <style type="text/css"> body { behavior: url(/style-v2/csshover3.htc); } </style> <![endif]-->
</head><!--
Note: Editing website content is documented at:
https://www.boost.org/development/website_updating.html
-->

<body>
  <div id="heading">
    <!--#include virtual="/common/heading.html" -->
  </div>

  <div id="body">
    <div id="body-inner">
      <div id="content">
        <div class="section" id="intro">
          <div class="section-0">
            <div class="section-title">
              <h1>Boost Test Policies and Protocols</h1>
            </div>

            <div class="section-body">
              <p>The Boost libraries are intended to be both reliable and
              portable. Every experienced programmer knows that means each
              library must be tested against a suitable number of test cases,
              on a wide range of platforms, and then tested again (regression
              tested) every time a change is made and before every
              release.</p>

              <p>"Quality assurance based on a wide range of targeted tests"
              as one of the key answers to C.A.R Hoare's question "How did
              software get so reliable without proof."</p>

              <h2>Regression test</h2>

              <p>Boost uses an automatic <a href=
              "/doc/tools/regression/index.html">regression
              test suite</a> which generates HTML <a href=
              "/development/testing.html#RegressionTesting">compiler status
              tables</a>.</p>

              <h2>Test Policy</h2>

              <h3>Required</h3>

              <ul>
                <li>Every Boost library should supply one or more suitable
                test programs to be exercised by the Boost <a href=
                "/doc/tools/regression/index.html">regression
                test suite</a>. In addition to the usual compile-link-run
                tests expecting successful completion, compile-only or
                compile-and-link-only tests may be performed, and success for
                the test may be defined as failure of the steps.</li>

                <li>Test program execution must report errors by returning a
                non-zero value. They may also write to stdout or stderr, but
                that output should be relatively brief. Regardless of other
                output, a non-zero return value is the only way the
                regression test framework will recognize an error has
                occurred. Note that test programs to be included in the
                status tables must compile, link, and run quickly since the
                tests are executed many, many, times.</li>

                <li>Libraries with time consuming tests should be divided
                into a fast-execution basic test program for the status
                tables, and a separate full-coverage test program for
                exhaustive test cases. The basic test should concentrate on
                compilation issues so that the status tables accurately
                reflect the library's likelihood of correct compilation on a
                platform.</li>

                <li>If for any reason the usual test policies do not apply to
                a particular library, an alternate test strategy must be
                implemented.</li>

                <li>A <a href=
                "/doc/tools/regression/index.html#Adding_new_test">
                Jamfile</a> to drive the regression tests for the
                library.</li>
              </ul>

              <h3>Optional (but highly recommended)</h3>

              <p>The <a href="/doc/libs/release/libs/test/index.html">Boost
              Test Library</a> provides many useful components which ease the
              construction of test programs.</p>

              <ul>
                <li>Use the library's <a href=
                "/doc/libs/release/libs/test/doc/html/boost_test/testing_tools.html">
                Test Tools</a> for the construction of simple test programs
                that do not need much structure.</li>

                <li>Use the library's <a href=
                "/doc/libs/release/libs/test/doc/html/boost_test/tests_organization.html">Unit
                Test Framework</a> for the construction of more complex test
                programs that need to be structured into individual tests and
                test suites.</li>
              </ul>

              <h2>Suggested Protocol for Fixing Bugs or Adding Features.</h2>

              <ul>
                <li>First, add regression test cases that detects the bug or
                tests the feature. Sometimes adding one case suggests similar
                untested cases, and they are added too.</li>

                <li>Second, for bugs, run the regression test and verify that
                the bug is now detected.</li>

                <li>Third, then, and only then, fix the bug or add the
                feature.</li>

                <li>Finally, rerun the full regression tests - sometimes the
                change breaks something else.</li>
              </ul>

              <h2>History</h2>

              <p><a href=
              "/doc/tools/regression/index.html#History">See
              Regression Test History</a>.</p>

              <h2>Acknowledgements</h2>

              <p>Written by Beman Dawes. Jens Maurer, Paul Moore, Gary Powell
              and Jeremy Siek contributed helpful suggestions.</p>
              <hr />
            </div>
          </div>
        </div>
      </div>

      <div id="sidebar">
        <!--#include virtual="/common/sidebar-common.html" -->
        <!--#include virtual="/common/sidebar-development.html" -->
      </div>

      <div class="clear"></div>
    </div>
  </div>

  <div id="footer">
    <div id="footer-left">
      <div id="revised">
        <p>Revised $Date$</p>
      </div>

      <div id="copyright">
        <p>Copyright Beman Dawes 2001.</p>
      </div><!--#include virtual="/common/footer-license.html" -->
    </div>

    <div id="footer-right">
      <!--#include virtual="/common/footer-banners.html" -->
    </div>

    <div class="clear"></div>
  </div>
</body>
</html>
